Relevance & Research Question: Although some research efforts have devoted to the comparison of survey findings between probability- and nonprobability-based Web survey samples, different types of nonprobability-based samples have not been thoroughly examined. In this study, we compare the data quality between online access panel sample and river sample. Online panel refers to a pre-recruited and profiled pool of respondents. Email addresses of the panelists are obtained and surveys are sent to them through emails. River sample is a pool of respondents that are obtained through banners, ads, or promotions. Anyone can click on them and subsequently respond to a survey. Respondents are not pre-recruited or profiled.

Methods & Data: In this study, we present findings from a study comparing survey responses between SurveyMonkey Audience (online panel) and Tapjoy (river sample). Three surveys (52, 29, and 19 questions respectively) are administered to both samples. The data quality indicators examined include speeding, open end validity, response rounding, straight lining, trapping, knowledge, and sensitive questions. Propensity score weighting is used to adjust the two samples in order to account for the effect of differential sample compositions. Since all river samples are mobile respondents, we also analyze the data by extracting the mobile respondents from the panel.

Results: For the 52 question version, about one third of the questions examined show significant difference between two samples. When restrict the comparison to mobile respondents only, the significant differences drop to about only 20% of the items. For the most part, the online panel produces superior quality data than river sample. Similar patterns are found for the 29 and 19 question versions. Also, as the survey length reduces, the difference between the two samples also decreases. This suggests that lengthy survey can cause poor data quality, particularly for the river sample.

Added Value: Understanding the difference between online panel and river samples is of practical importance survey researchers and practitioners, when choosing the sample sources for their studies and determining whether it is appropriate to blend sample from different sources.